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The  1991  Acquisition  Research  Symposium  is  the  latest  in  a  series  of  conferences 
begun  in  1972.  These  Symposia  offer  a  dynamic  forum  for  dialogue  among  key  professionals 
working  on  vital  issues  facing  the  acquisition  community.  Attendees  include  senior  officials, 
program  managers,  staff  officers,  and  researchers  from  the  Department  of  Defense,  federal 
civilian  agencies,  academia,  and  industry. 

This  year’s  theme  reflects  the  future  innovation  and  implementation  in  the  acquisition 
process.  "Acquisition  for  the  Future  -  Imagination,  Innovation,  and  Implementation"  is  the 
prevailing  theme  discussed  and  examined  throughout  this  publication.  The  papers  included 
cover  the  latest  research  and  development  as  documented  by  individuals  involved  in  the  many 
aspects  of  the  acquisition  process. 

We  invite  you  to  take  advantage  of  this  publication,  which  expands  upon  Symposium 
presentations  and  introduces  new  authors  and  topics.  Please  note  that  the  views  expressed  are 
those  of  the  authors  and  do  not  necessarily  reflect  the  views  of  the  organization  with  which 
they  are  associated. 


CONTRACTING  IN  THE 
SOFTWARE  ENGINEERING  CRISIS 


Mr.  Dennis  M.  Bauman  and  Mr.  Albert  E.  Jensen,  The  Naval  Ocean  Systems  Center 


ABSTRACT 

The  Department  of  Defense  (DoD)  is 
currently  experiencing  extremely  serious 
problems  with  Software  Engineering.  As 
weapons  systems  become  increasingly 
more  dependent  upon  the  software 
programs  which  they  employ,  it  becomes 
evermore  apparent  that  DoD  projects  are 
consistently  late  and  over  budget  because 
of  software.  To  this  end,  software  has 
become  the  dominant  risk  to  cost  and 
schedule  and  has  caused  the  demise  of 
more  than  a  few  DoD  development 
projects. 

The  DoD  repeatedly  demonstrates 
requisite  experience  to  efiQciently  manage 
the  development  of  hardware.  Hardware 
development  projects,  for  which  the  DoD 
readily  implements  tried  and  proven 
practices  to  systematically  address  all 
aspects  of  the  development,  are  relatively 
risk  free  at  their  onset. 

Software  development  projects,  on  the 
other  hand,  are  oftentimes  im’tiated 
without  the  benefit  of  similarly 
institutionalized  methodologies  and 
practices.  Because  an  understanding  of 
proper  Software  Engineering  practice  is 
only  now  developing,  standard, 
well-accepted  measures  do  not  yet  exist. 
The  DoD  seems  only  to  be  scratching  at 
the  surface  of  state-of-the-art  software 
engineering  and  herein  lies  the  basis  of 
the  problem. 


INTRODUCTION 

Experience  at  the  Naval  Ocean  Systems 
Center  has  demonstrated  that  Software 
Engineering  plays  an  ever  increasing,  key 
role  in  our  ability  to  be  successful, 
responsive  to  sponsor  requirements,  and 
competitive.  Recent  irmovations  in  the 
Software  Engineering  discipline  promise  to 
reduce  risks  associated  with  this  type  of 
development  if  we  are  able  to  effectively 
adopt  and  utilize  them.  The  Naval  Ocean 
Systems  Center  Technical  Director  has 
made  a  major  commitment  of  Center 
resources  by  establishing  the  Software 
Engineering  Process  Office,  and 
implementing  plans  to  eventually  reach 
higher  levels  of  process  maturity.  Only 
through  proactive  measures  such  as  this 
can  we,  as  a  Center,  continue  to  meet  our 
mission  in  this  vital  area.  In  fact,  the 
Software  Engineering  Crisis  threatens  our 
reputation  and  thus,  our  continued  work 
in  Navy  systems. 

The  approach  adopted  by  the  Technical 
Director  only  addresses  part  of  the 
problem,  perhaps  a  minor  part  of  the 
problem.  Most  of  the  software  developed 
at  this  Center  is  developed  by  support 
contractors.  The  average  locally  based 
support  contractor  is  incapable,  without 
substantial  initial  and  sustained 
investment,  and  sharply  increased  skill 
levels,  of  performing  the  state-of-the-art 
Software  Engineering  needed  to  reduce 
the  risks  associated  with  software 
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development  projects.  In  order  for  the 
Naval  Ocean  Systems  Center  to  be  able  to 
take  advantage  of  these  new  Software 
Engineering  Processes,  our  support 
contractors  must  be  motivated  to  become 
partners  with  us  by  adopting  the  same 
philosophies  and  taking  similar  proactive 
measures  to  improve  our  Software 
Engineering  capabilities.  Neither  can  do 
it  alone.  As  contractors  improve  their 
ability  to  meet  our  needs  for  quality 
software,  our  ability  to  serve  the  national 
interest  will  be  improved  by  awarding 
contracts  to  those  with  the  best  capability. 
To  provide  this  motivation,  we  must  give 
our  contractors  an  opportunity  to  compete 
with  one  another  with  regard  to  modem 
Software  Engineering,  and  we  therefore 
need  the  tools  with  which  to  measure 
Software  Engineering  competency.  We 
have  theuL 

CAPABBUTY  ASSESSMENT 

The  Software  Engineering  Institute  (SEI) 
is  a  federally  funded  research  and 
development  center,  formed  in  1984  in 
response  to  the  need  for  advances  across 
all  phases  of  the  Software  Engineering 
process.  The  SEI  is  a  imit  of  Carnegie 
Mellon  University,  under  contract  with  the 
DoD.  Its  mission,  based  upon  an 
assumption  that  sound  engineering 
processes  lead  to  quality  software,  is  to 
influence  rapid  improvement  of  the  quality 
of  operational  software  in  mission-critical 
computer  systems,  to  accelerate  the 
reduction  to  practice  of  modem  Software 
Engineering  techniques  and  methods,  to 
promulgate  the  use  of  modem  techniques 
and  methods  throughout  the 
mission-critical  systems  community,  and  to 
establish  standards  of  excellence  for 
Software  Engineering  practice. 


ASSESSMENT  TOOL 

The  SEI  Technical  Report  "A  Method  for 
Assessing  the  Software  Engineering 
Capability  of  Contractors"  dated 
September  1987  provides  us  with  a  tool,  as 
its  title  implies,  with  which  to  assess  the 
Software  Engineering  Process  Maturity  of 
our  contractors.  We  found  that  the  SEI 
assessment  instmment  can  be  used  to 
facilitate  objective  and  consistent 
assessments  of  the  ability  of  potential 
DoD  contractors  to  develop  software  in 
accordance  with  modern  Software 
Engineering  methods.  This  assessment 
instnunent  is  basically  a  questionnaire 
calling  only  for  yes  or  no  answers  to 
questions  based  on  the  following  premises: 

-  The  quality  of  a  software  product 
stems,  in  large  part,  from  the  quality  of 
the  process  used  to  create  it. 

-  Software  engineering  is  a  process  that 
can  be  managed,  measured,  and 
progressively  improved. 

-  The  quahty  of  a  software  process  is 
affected  by  the  technology  used  to  support 
it. 

-  The  level  of  technology  used  in 
software  engineering  should  be 
appropriate  to  the  maturity  of  the  process. 

Software  products  developed  by 
contractors  for  DoD  use  are  acquired 
under  contracts  invoking 
DOD-STD-2167A,  Defense  System 
Software  Development,  as  tailored  for 
each  contract. 

TTie  SEI  questionnaire  is  arranged  so  that 
the  capability  to  perform  software 
engineering  is  divided  into  three  areas: 
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-  Organization  and  resource 
management, 

-  Software  engineering  process 
and  its  management,  and 

-  Tools  and  technology. 

To  provide  a  structure  for  assessment,  five 
levels  of  process  maturity  and  two  stages 
of  technology  advancement  have  been 
postulated: 

Process  Maturity  Levels: 

1  -  Initial:  The  initial  environment  has 
ill-defined  procedures  and  controls.  The 
organization  does  not  consistently  apply 
software  engineering  management  to  the 
process,  nor  does  it  use  modem  tools  and 
technology.  Level  1  organizations  may 
have  serious  cost  and  schedule  problems. 

2  -  Repeatable:  At  Level  2,  the 
organization  has  generally  learned  to 
manage  costs  and  schedules,  and  the 
process  is  now  repeatable.  The 
organization  uses  standard  methods  and 
practices  for  managing  software 
development  activities  such  as  cost  and 
estimating,  scheduling,  requirements 
changes,  code  changes,  and  status  reviews. 

3  -  Defined:  In  Level  3,  the  process  is 
well  characterized  and  reasonably  well 
understood.  The  organization  defines  its 
process  in  terms  of  software  engineering 
standards  and  methods,  and  it  has  made  a 
series  of  organizational  and 
methodological  improvements.  These 
specifically  include  design  and  code 
reviews,  training  programs  for 
programmers  and  review  leaders,  and 
increased  organizational  focus  on  software 
engineering.  A  major  improvement  in  this 
phase  is  the  establishment  and  staffing  of 


a  Software  Engineering  Process  Group 
that  fooises  on  the  software  engineering 
process  and  the  adequacy  with  which  it  is 
implemented, 

4  -  Managed:  In  Level  4,  the  process  is 
not  only  understood  but  it  is  quantified, 
measured,  and  reasonably  well  controlled. 
The  organization  typically  bases  its 
operating  decisions  on  quantitative  process 
data,  and  conducts  extensive  analyses  of 
the  data  gathered  during  software 
engineering  reviews  and  tests.  Tools  are 
used  increasingly  to  control  and  manage 
the  design  process  as  well  as  to  support 
data  gathering  and  analysis.  The 
organization  is  learning  to  project 
expected  errors  with  reasonable  accuracy. 

5  -  Optimized:  At  Level  5, 

organizations  have  not  only  achieved  a 
high  degree  of  control  over  their  process, 
they  have  a  major  focus  on  improving  and 
optimizing  its  operation.  This  includes 
more  sophisticated  analyses  of  the  error 
and  cost  data  gathered  during  the  process 
as  well  as  the  introduction  of 
comprehensive  error  cause  analysis  and 
prevention  studies.  The  data  on  the 
process  are  used  iteratively  to  improve  the 
process  and  achieve  optimum 
performance. 

Software  Technology  Stages: 

A  -  Inefficient:  Multiple 

implementations  may  be  avaialble  and  the 
practice  may  be  in  widespread  use,  but  the 
technology  is  no  longer  effective.  An 
organization  that  primarily  erhploys 
inefficient  software  development 
technology  is  likely  to  be  ineffective  in 
developing  software.  Moreover,  at  this 
technology  stage  some  important  software 
engineering  practices  are  not  practical  in 
large  complex  developments. 
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B  -  Basic:  Multiple  implementations 
are  available,  and  they  have  been 
demonstrated  to  be  effective.  An 
organization  that  primarily  employs  basic 
software  development  technologies  is 
likely  to  be  moderately  effective  and, 
depending  upon  the  maturity  of  its 
process,  reasonably  consistent  in  its 
performance. 

SEI  Guidance 

The  SEI  offers  guidance  for  assessing  the 
capability  of  contractors,  using  the 
assessment  instrument.  Such  assessments 
may  be  conducted  either  in  the 
pre-solicitation  qualification  process,  in 
the  formal  source  selection  process,  or 
both.  This  Software  Capability  Evaluation 
(SCE)  method  should  be  used  to  augment 
the  many  steps  currently  involved  in 
source  selection.  However,  the 
effectiveness  of  a  SCE  is  critically 
dependent  on  the  process  used  in  the 
assessment  and  on  the  background  and 
training  of  the  personnel  conducting  it. 
Information  contained  in  the  document 
itself  providesthe  SEI  guidance  for  its  use: 

-  When  used  as  part  of  the  formal  DoD 
systems  acquisition  process,  the  questions 
are  furnished,  for  information  purposes,  to 
potential  bidders  with  the  Request  for 
Proposal  (RFP). 

-  Answers  to  the  assessment  questions 
are  not  submitted  with  the  proposal,  but 
are  provided  to  an  assessment  team  that 
visits  each  competing  contractor  during 
the  proposal  evaluation  period. 

-  Several  days  of  classroom  instruction 
must  be  afforded  each  of  the  competing 
contractors,  to  review  the  assessment 
questioimaire  in  detail  and  discuss  the 


materials  and  support  tools  that  should  be 
available  to  demonstrate  performance  for 
each  question. 

-  The  Government  assessment  team  will 
visit  each  competing  contractor  during  the 
evaluation  period.  Several  major  software 
development  projects,  as  agreed  to  by  the 
contractor  and  the  assessment  team,  will 
be  assessed.  A  period  of  3  to  4  days  is 
needed  to  review  the  questions,  obtain 
and  discuss  back-up  material,  demonstrate 
support  tools,  and  present  conclusions.  A 
single  assessment  team  should  be  used  to 
visit  all  of  the  competing  firms  to  assure 
consistent  interpretation  of  both  the 
questions  and  the  results. 

-  The  assessment  team  must  have  a  mix 
of  talents.  A  minimum  of  four 
experienced  professionals  are  required, 
including  those  knowledgeable  in  the 
software  development  process,  the 
technology,  the  application  area,  and  the 
specific  procurement.  All  team  members 
must  have  been  trained  in  the  SEI  SCE 
process.  This  training  is  available  in  a 
3-day  course  of  instruction  offered  by  the 
SEI  at  Carnegie  Mellon  University, 


-  At  the  conclusion  of  each  assessment, 
the  contractor's  management  is  informed 
of  the  findings  and  given  an  opportunity  to 
offer  evidence  to  refute  any  disputed 
findings  and  to  explain  their  plans  for 
process  improvement. 


-  The  results  of  each  assessment  are 
made  available  to  the  Source  Selection 
officials  for  consideration  prior  to  contract 
award. 


Following  the  SEI  guidance,  as  above,  will 
provide  a  thorough  assessment  of  (he 
software  engineering  capabilities  and 
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process  maturity  of  MJ  competing 
contractors.  However,  the  following 
points  should  be  considered: 

-  Costs  involved  with  the  above 
implementation  could  soar.  Training  and 
on-site  assessment  for  a  single  contractor 
may  be  well  in  excess  of  $20,000, 
depending  upon  location,  i.e.,  local 
contractor,  or  one  which  is  located 
somewhere  between  San  Diego  and  the 
East  Coast.  Furthermore,  there  is  an 
average  of  seven  respondents  to  every 
solicitation  issued  by  the  Naval  Ocean 
Systems  Center. 

-  The  above  implementation  process 
could  lead  to  contract  award  to  an 
unqualified  firm  if  none  of  the  competing 
contractors  are  at  the  software  process 
maturity  level  required  to  support  the 
requirements  of  the  contract. 

EMERGENT  NEED 

In  the  light  of  the  current 
Communications  Department  and 
Cont.„cts  policy  to  move  away  from  the 
large,  omnibus  type  contracts  to  smaller 
and  more  project  specific  contracts,  we  in 
the  Operational  Systems  Branch,  Code 
833,  of  the  Submarine  Communications 
Division  were  faced  with  an  emergent 
need  for  contract  support  for  several  of 
our  projects.  We  determined  that  the 
contract  support  needed  covered  a  broad 
range  of  disciplines,  but  that  the 
predoramant  need  was  for  efficient 
software  engineering  which  would  enable 
significant  risk  reduction  in  current  and 
future  development  projects.  More 
specifically,  our  need  was  determined  to 
be  contractor  support  which  could 
immediately  respond  with  software 
engineering  capability  commensurate  with 


SEI  Level  2,  or  higher  process  maturity. 
We  decided  to  invoke  the  SEI  Software 
Capability  Evaluation  (SCE)  method  in  a 
solicitation  and  source  selection  pro  'ess. 

OUR  APPROACH 

Since  the  scope  of  the  support  contract 
would  cover  a  wide  range  of  disciplines. 
We  were  concerned  that  a  single 
contractor,  capable  of  modern  software 
engineering  practices,  may  not  have  the 
necessary  background  and  experience 
needed  to  adequately  support  the 
remaining  requirements.  For  this  reason, 
we  prepared  a  synopsis  for  publication  in 
the  Commerce  Business  Daily  (CBD) 
which  encouraged  contractor  teaming. 

Because  of  the  considerations  above,  i.e., 
high  cost  and  the  possibility  of  gaining 
unqualified  contractor  support,  the  SEI 
assessment  methodology  had  to  be 
tailored.  Therefore,  we  developed  a 
Source  Selection  Plan  which  provided  the 
necessary  tailoring  to  the  SEI  guidance  for 
using  the  assessment  document,  and 
clearly  stated  the  additional  requirements 
to  be  incorporated  into  the  solicitation: 

-  A  prerequisite  that  any  respondent  to 
the  RFP  demonstrate  that  they  are 
curre  .tly  performing  software  engineering 
practices  at  the  SEI  Level  2,  or  higher 
process  maturity,  was  established. 

-  To  determine  the  level  of  software 
process  maturity,  the  SEI  assessment 
document  was  included  with  the.  RFP; 
each  offeror  was  required  to  perform  a 
"self-assessment".  Results  of  the 
self-assessment  were  submitted  with  the 
proposal;  however,  this  was  not  used  in 
conjunction  with  the  technical  evaluation 
for  scoring  purposes. 
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-  Technical  and  cost  proposals  were 
evaluated  to  determine  the  relative 
ranking  of  all  offers  with  respect  to  the 
"greatest  value  to  the  Government". 
However,  prior  to  contract  award,  the 
Government  did  perform  an  on-site 
validation  of  the  contractor's  SEI 
self-assessment.  The  validation  was 
performed  by  a  qualified,  SEI  trained 
team  of  professionals.  The  RFP  stated 
that  in  the  event  that  a  contractor  in  line 
for  award,  technical  and  cost  considered, 
failed  to  demonstrate  a  current  SEI  Level 
2  or  higher  process  maturity,  the  next 
offeror  in  line  for  award  woiild  undergo 
this  on-site  validation,  and  so  forth  until 
the  otherwise  qualified  contractor,  meeting 
the  prerequisite  SEI  level  of  process 
maturity  was  determined. 

To  strengthen  the  technical  proposals,  and 
to  ensure  that  the  selected  contractor 
would  continue  to  perform  at  the  leading 
edge  of  software  engineering,  we  required 
each  competing  contractor  to  submit  a 
"Software  Standards  and  Procedures  Plan" 
as  a  part  of  their  proposal.  In  the  RFP, 
we  specified  the  criteria  to  be  presented  in 
the  plan.  This  criteria  consisted  simply  of 
those  individual  elements  commemurate 
with  SEI  Level  2  process  maturity.  In 
order  to  ensure  compliance,  this  plan  was 
heavily  weighted  within  the  overall 
technical  evaluation.  And  finally,  we 
required  that  this  plan  become  binding 
upon  the  contractor  for  all  software  work 
to  be  performed  under  the  contract. 

The  importance  of  the  technical  proposals 
was  established  in  the  source  selection 
process  by  setting  a  relatively  high 
technical  to  cost  ratio  in  the  source 
selection  plan.  Our  approach  to 
contractor  source  selection  for  work  on 
"Airborne  Submarine  Commum’cations" 


projects  was  successful  in  gaining  the  level 
of  engineering  competence  needed.  By 
design,  our  approach  led  to  contract  award 
only  to  a  qualified,  SEI  Level  2  or  higher 
firm.  It  did;  however,  exclude  from 
competition,  any  firm  which  may  be 
presently  SEI  Level  1  albeit  very  close  to 
SEI  Level  2  performance.  We  found  our 
approach  to  be  highly  cost  effective  since 
we  needed  to  conduct  only  one  on-site 
validation  of  the  SEI  self-assessment.  Our 
approach  readily  demonstrated  our 
intention,  as  evidenced  by  the  unusual 
number  of  contractor  questions,  cries  of 
unfairness,  and  one  formal  protest,  to 
insure  that  the  contract  award  be 
competed  primarily  on  technical  issues. 
After  all,  the  SEI  Level  2  process  maturity 
has  not  previously  been  a  prerequisite  to 
contract  award.  Our  approach  to 
contractor  selection  provided  the  following 
experiences; 

-  We  received  four  proposals  to  our 
solicitation,  each  representing  contractor 
teaming  arrangements  with  a  single  prime 
contractor. 

-  One  offer  was  eliminated  initially, 
based  on  a  very  weak  technical  proposal. 
This  left  three  offers  in  the  technically 
competitive  range. 

-  Evaluation  of  three  Best  and  Final 
Offers  reinforced  the  original  ranking; 
however,  eliminated  one  offeror  from  the 
technically  competitive  range. 

-  The  contractor  in  line  for  award, 
based  on  the  "Greatest  Value  'to  the 
Government"  as  determined  by  technical 
and  cost  evaluations,  was  also  determined 
to  be  currently  operating  at  the 
prerequisite  level  of  software  process 
maturity. 


16 


-  There  was  one  formal  protest  which 
was  withdrawn  following  technical 
debriefings  and  clarification  of  the 
Government  selection  process. 

CONCLUSION 

In  conclusion,  the  SEI  guidance  for  using 
their  Technical  Report,  "A  Method  for 
Assessing  the  Software  Engineering 
Capability  of  Contractors",  dated 
September  1987,  can  be  extremely  time 


consuming,  prohibUively  expensive,  and 
could  lead  to  contract  award  to  an 
unqualified  firm.  The  Naval  Ocean 
Systems  Center’s  approach  to 
implementing  this  assessment  instrument 
in  the  solicitation  and  source  selection 
process  offers  eaq>cdiency,  provides  cost 
control  over  the  process,  and  excludes  all 
potentially  tmqualified  offerors.  However, 
the  risk  of  contractor  protest  should  be 
considered  under  the  latter  approach. 
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